Felix: Scaling up Global Statistical Information Extraction Using an Operator-based Approach

To support the next generation of sophisticated information extraction (IE) applications, several researchers have proposed frameworks that integrate SQL-like languages with statistical reasoning. While these frameworks demonstrate impressive quality on small IE tasks, they currently do not scale to enterprise-sized tasks. To enable the next generation of IE, a promising approach is to improve the scalability and performance of such statistical frameworks. Our technical observation is that many IE subtasks, such as coreference resolution or classification, can be solved by specialized algorithms that achieve both high quality and high performance. In contrast, current general-purpose statistical inference approaches are oblivious to these subtasks and so use a single algorithm independent of the subtask that they are performing. We present Felix, in which programs are expressed in a general statistical inference language (called Markov logic). Felix first breaks the program into a handful of subtasks, which can then be executed using predefined operators, i.e., statistical algorithms. A key challenge Felix faces is to decide whether or not to materialize intermediate results from the operators. To attack this challenge, Felix uses a cost-based approach that relies on the RDBMS optimizer. Using all of our techniques, we show that Felix efficiently processes global IE programs on large real-world datasets while prior approaches crash or take days. Felix, in turn, is able to execute programs that achieve higher quality than state-of-the-art IE approaches on three real-world datasets.